Image processing method, image processing device, game device, and program

ABSTRACT

The present invention provides an image processing method, wherein a computer executes: an acquisition step S10 of acquiring image information indicating, on a per-pixel basis, distance information indicating the distance from a camera and color information; a determination step S20 of determining, on a per-pixel basis and on the basis of the distance information of individual pixels, settings of a modulation filter that converts the color information of the individual pixels to modulate an image into the style of a painting; and a conversion step S30 of converting the color information on a per-pixel basis on the basis of the settings of the modulation filter determined for the individual pixels.

TECHNICAL FIELD

The present invention relates to image processing methods, imageprocessing devices, game devices, and programs.

BACKGROUND ART

As modulation filters for modulating an image into the style of apainting, the symmetric nearest neighbor (SNN) filter, the medianfilter, the Kuwahara filter, etc. are well known. These modulationfilters modulate an entire image into the style of a painting byconverting color information of each pixel on the basis of colorinformation of other pixels (e.g., other pixels around that pixel).

Patent Literatures 1 and 2 disclose technologies for generating an imagein which fog (fog, cloud, steam, haze, dirt, dust, smoke, tornado, dew,etc.) is applied to an original image, wherein an a value, which is aparameter for controlling the fog density in accordance with thedistance from the viewpoint (virtual camera), is changed so as toincrease the fog density as the distance from the viewpoint increases.

CITATION LIST Patent Literature

PTL 1

Japanese Unexamined Patent Application, Publication No. 2006-318386

PTL 2

Japanese Unexamined Patent Application, Publication No. 2006-318389

SUMMARY OF INVENTION Technical Problem

In the above-described ordinary processing for modulating an image intothe style of a painting by using a modulation filter, color informationof all pixels is converted uniformly by the same degree (e.g., thedegree of blurring or the degree of bleeding). The adjustment of thedegree is realized, for example, by adjusting the number of otherpixels, the weights of color information of other pixels, etc. that arereferred to when converting color information.

In the case of the above-described ordinary processing, although it ispossible to generate an image that is painting-like in its entirety, itis not possible to generate an image in which both edge clarity andpainting-like qualities of predetermined content to be rendered arerealized. Patent Literatures 1 and 2 do not disclose or suggest thisproblem to be addressed or a solution therefor concerning a modulationfilter that modulates an image into the style of a painting.

It is an object of the present invention to render an image whilerealizing both edge clarity and painting-like qualities of predeterminedcontent to be rendered.

Solution to Problem

The present invention provides an image processing method,

wherein a computer executes:

-   -   an acquisition step of acquiring image information indicating,        on a per-pixel basis, distance information indicating the        distance from a camera and color information;    -   a determination step of determining, on a per-pixel basis and on        the basis of the distance information of individual pixels,        settings of a modulation filter that converts the color        information of the individual pixels to modulate an image into        the style of a painting; and    -   a conversion step of converting the color information on a        per-pixel basis on the basis of the settings of the modulation        filter determined for the individual pixels.

The present invention provides an image processing device including:

an acquisition unit that acquires image information indicating, on aper-pixel basis, distance information indicating the distance from acamera and color information;

a determination unit that determines, on a per-pixel basis and on thebasis of the distance information of individual pixels, settings of amodulation filter that converts the color information of the individualpixels to modulate an image into the style of a painting; and

a conversion unit that converts the color information on a per-pixelbasis on the basis of the settings of the modulation filter determinedfor the individual pixels.

The present invention provides a program for causing a computer tofunction as:

an acquisition means for acquiring image information indicating, on aper-pixel basis, distance information indicating the distance from acamera and color information;

a determination means for determining, on a per-pixel basis and on thebasis of the distance information of individual pixels, settings of amodulation filter that converts the color information of the individualpixels to modulate an image into the style of a painting; and

a conversion means for converting the color information on a per-pixelbasis on the basis of the settings of the modulation filter determinedfor the individual pixels.

The present invention provides a game device including: an inputaccepting unit that accepts an operational input from a player;

a player-character control unit that manages the position andorientation of a player character in a virtual three-dimensional space;

a non-player-character control unit that manages the position andorientation of a non-player character in the virtual three-dimensionalspace;

a camera control unit that manages the position and orientation of avirtual camera in the virtual three-dimensional space;

a rendering unit that renders the virtual three-dimensional spacecaptured by the virtual camera as an image on a two-dimensional plane onthe basis of the positions and orientations of the player character, thenon-player character, and the virtual camera in the virtualthree-dimensional space; and

a display control unit that causes an output device to display the imageon the two-dimensional plane,

wherein the rendering unit includes:

-   -   an acquisition unit that acquires image information indicating,        on a per-pixel basis, distance information indicating the        distance from the virtual camera and color information;    -   a determination unit that determines, on a per-pixel basis and        on the basis of the distance information of individual pixels,        settings of a modulation filter that converts the color        information of the individual pixels to modulate an image into        the style of a painting; and    -   a conversion unit that converts the color information on a        per-pixel basis on the basis of the settings of the modulation        filter determined for the individual pixels.

Advantageous Effects of Invention

The present invention makes it possible to render an image whilerealizing both edge clarity and painting-like qualities of predeterminedcontent to be rendered.

BRIEF DESCRIPTION OF DRAWINGS

The abovementioned object, other objects, features and advantages willbecome more apparent in view of preferred embodiments described belowand the following accompanying drawings.

FIG. 1 is a diagram showing an example hardware configuration of animage processing device according to the present embodiment.

FIG. 2 is an example functional block diagram of the image processingdevice according to the present embodiment.

FIG. 3 explains conversion processing according to the presentembodiment.

FIG. 4 schematically shows an example of reference area informationaccording to the present embodiment.

FIG. 5 shows content defined in the example of reference areainformation according to the present embodiment.

FIG. 6 is a flowchart showing an example flow of processing by the imageprocessing device according to the present embodiment.

FIG. 7 is a flowchart showing an example flow of processing by the imageprocessing device according to the present embodiment.

FIG. 8 is a flowchart showing an example flow of processing by the imageprocessing device according to the present embodiment.

FIG. 9 is an example functional block diagram of a game device accordingto the present embodiment.

FIG. 10 is an example functional block diagram of a rendering unitaccording to the present embodiment.

DESCRIPTION OF EMBODIMENTS First Embodiment

First, an overview of processing by an image processing device accordingto this embodiment will be described. In the image processing deviceaccording to this embodiment, upon acquiring image informationindicating “distance information indicating the distance from a camera”and “color information” on a per-pixel basis, the settings of “amodulation filter that converts the color information of individualpixels to modulate an image into the style of a painting” are determinedon a per-pixel basis on the basis of the distance information of theindividual pixels. The settings that are determined here are the degreesof conversion (e.g., the degrees of blurring or the degrees of bleeding)of the color information of the individual pixels. The degrees ofmodulation of the image are adjusted by adjusting the settings.Furthermore, the image processing device converts the color informationon a per-pixel basis on the basis of the settings of the modulationfilter determined for the individual pixels.

As described above, with the image processing device according to thisembodiment, it is possible to set the degrees of conversion (e.g., thedegrees of blurring or the degrees of bleeding) of the color informationof individual pixels on a per-pixel basis and to convert the colorinformation of the individual pixels as set on a per-pixel basis. Thus,with the image processing device according to this embodiment, it ispossible to vary the degrees of conversion partially within a singleimage. That is, with the image processing device according to thisembodiment, it is possible to generate a single image in which a portionwith a high degree of conversion, a portion with a low degree ofconversion, a portion that has not been converted, etc. coexist.Accordingly, with the image processing device according to thisembodiment, it becomes possible to modulate an image into the style of apainting while realizing both edge clarity and painting-like qualitiesof predetermined content to be rendered.

Next, the configuration of the image processing device according to thisembodiment will be described. First, an example hardware configurationof the image processing device will be described. The functional unitsprovided in the image processing device according to this embodiment arerealized by an arbitrary combination of hardware and software mainlyincluding an arbitrary computer having a central processing unit (CPU),a memory, programs loaded into the memory, a storage unit that storesthe programs (which can store programs already stored at the time ofshipping of the device as well as programs stored in a storage medium,such as a compact disc (CD), or downloaded from a server or the like onthe Internet), such as a hard disk, and a network connection interface.Furthermore, it would be understood by a person skilled in the art thatthere are various modifications concerning the method of realization andthe device.

FIG. 1 is a block diagram showing an example hardware configuration ofthe image processing device according to this embodiment. As shown inFIG. 1, the image processing device includes a processor 1A, a memory2A, an input/output interface 3A, a peripheral circuit 4A, and a bus 5A.The peripheral circuit 4A includes various modules. The image processingdevice need not include the peripheral circuit 4A. Note that the imageprocessing device may be configured of a plurality of physically and/orlogically separated devices. In this case, each of the plurality ofdevices may include the hardware configuration described above.

The bus 5A is a data transmission path that allows the processor 1A, thememory 2A, the peripheral circuit 4A, and the input/output interface 3Ato mutually send and receive data. The processor 1A is a computationalprocessing device, such as a CPU or a graphics processing unit (GPU).The memory 2A is a memory such as a random access memory (RAM) or a readonly memory (ROM). The input/output interface 3A includes an interfacefor obtaining information from an input device, an external device, anexternal server, an external sensor, etc., an interface for outputtinginformation to an output device, an external device, an external server,etc., and so forth. The input device is, for example, a controller, akeyboard, a mouse, a microphone, a pointing device, a touchscreen, aphysical button, or a camera. The output device is, for example, adisplay, a speaker, a printer, or a mailer. The processor 1A can issueinstructions to the individual modules and can perform computations onthe basis of the results of corresponding computations.

Next, the functional configuration of the image processing deviceaccording to this embodiment will be described. FIG. 2 shows an examplefunctional block diagram of an image processing device 10. As shown inthe figure, the image processing device 10 includes an acquisition unit11, a determination unit 12, and a conversion unit 13.

The acquisition unit 11 acquires image information, which is informationconcerning an image and indicates color information and distanceinformation on a per-pixel basis.

The image is, for example, an image indicating a scene in a game. Morespecifically, the image is obtained by capturing an image of(photographing) a plurality of items to be rendered, whose positioninformation in a three-dimensional space is managed, from a virtualcamera set at a predetermined viewpoint and subjecting the image totwo-dimensional perspective projection. Examples of the items to berendered include characters that appear in the game and objectsconstituting parts of the background (plants such as trees, structuressuch as houses, etc.), but there is no limitation to these examples. Asthe method of hidden surface removal at the time of rendering atwo-dimensional image, for example, depth buffering (Z-buffering) isused.

The color information indicates the colors of the items to be renderedrepresented by the individual pixels. The distance information indicatesthe distances from the camera (viewpoint) to the items to be renderedrepresented by the individual pixels.

The determination unit 12 determines the settings of a modulation filterthat converts the color information of individual pixels to modulate animage into the style of a painting (hereinafter simply referred to as a“modulation filter” depending on the case) on a per-pixel basis on thebasis of the distance information of the individual pixels.

The modulation filter is a filter that converts an image into the styleof a painting by converting the color information of each pixel on thebasis of the color information of the other pixels. Examples of themodulation filter include the symmetric nearest neighbor (SNN) filter,the median filter, and the Kuwahara filter, but there is no limitationto these examples.

An overview of processing using the modulation filter will be describedwith reference to FIG. 3. FIG. 3 shows a plurality of pixels. Here, theindividual pixels are identified by using numerals assigned to theindividual rows and letters assigned to the individual columns. Forexample, the pixel located on the row to which 1 is assigned and on thecolumn to which B is assigned will be referred to as a pixel 1B.

In the processing using the modulation filter, a reference area is setso as to include a pixel being subjected to processing. The referencearea may include either just the pixel being subjected to processing orthe pixel being subjected to processing as well as other pixels (e.g.,pixels located around the pixel being subjected to processing). In thecase of the example in FIG. 3, a pixel 3C is the pixel being subjectedto processing, and the hatched area is the reference area. The referencearea shown in the figure is a square centered at the pixel beingsubjected to processing and having a predetermined size. The referencearea includes pixels 2B to 2D, pixels 3B to 3D, and pixels 4B to 4D.Furthermore, in the processing using the modulation filter, the colorinformation of the pixel being subjected to processing is converted onthe basis of the color information of the pixels in the reference area.There are various methods for this conversion, and there is noparticular limitation in this embodiment. Note that although thereference area shown in the figure is a square, the shape of thereference area is not limited to this shape.

Next, the settings that are determined by the determination unit 12 on aper-pixel basis will be described. The settings that are determined bythe determination unit 12 are the degrees of conversion (e.g., thedegrees of blurring or the degrees of bleeding) of the color informationof the individual pixels. The degrees of modulation of an image areadjusted by adjusting the settings.

An example of a parameter for adjusting the degree of conversion of thecolor information of each pixel is the size of the reference area. Thedegree of blurring or the degree of bleeding increases and the degree ofpainting-likeness increases as the reference area becomes larger.Meanwhile, the degree of blurring or the degree of bleeding decreasesand the degree of painting-likeness decreases as the reference areabecomes smaller.

The determination unit 12 determines the size of the reference area thatis set to each pixel on the basis of the distance information of thatpixel and a predefined rule. The predefined rule may be a formula forcalculating the size of the reference area from the distance indicatedby the distance information, a table defining the relationship betweenthe distance indicated by the distance information and the size of thereference area, etc.

The size of the reference area can be expressed, for example, in termsof the number of pixels. In this case, the size of the reference areathat is determined by the determination unit 12 is greater than or equalto one pixel and less than or equal to M pixels. The value of M is anarbitrary value greater than 1.

Here, as an example of processing by the determination unit 12, thefollowing describes processing in which the determination unit 12determines the size of the reference area on a per-pixel basis on thebasis of reference area information and distance information, where thereference area information defines the size of the reference areas on aper-level basis for N different levels (N is an integer greater than orequal to 2) of the distance from the camera.

FIG. 4 shows an example of reference area information defining the sizesof reference areas for the individual levels of the distance from thecamera. The reference area information shown in the figure includeslevel numbers, distance ranges, thresholds, and formulas.

The level numbers are numbers individually assigned to a plurality oflevels and serve as information for identifying the plurality of levels.N=5, i.e., the distance from the camera is divided into five levels. Thedistance ranges indicate the individual distance ranges of the pluralityof levels, for example, in terms of upper limit values and lower limitvalues. The thresholds indicate the sizes of reference areas in the casewhere the distance is the greatest in the respective levels. The sizesof reference areas are expressed in terms of the numbers of pixels. Theformulas are equations for calculating the sizes L of reference areas.An operator determines the value of N as well as the distance ranges,thresholds, and formulas for the individual levels, and registersreference area information including the determined content in the imageprocessing device 10. Note that the specific examples of the value of Nas well as the distance ranges, thresholds, and formulas for theindividual levels, shown in the figure, are only examples, and there isno limitation to these examples.

FIG. 5 shows the relationship between the distance from the camera andthe size of the reference area as defined in the reference areainformation in FIG. 4. In the figure, the vertical axis indicates thesize of the reference area, and the horizontal axis indicates thedistance from the camera.

Since the threshold for level 1 is nine pixels according to FIG. 4, whenthe distance is the greatest in level 1 (at the boundary between level 1and level 2), the size of the reference area is nine pixels, as shown inFIG. 5.

Furthermore, since the formula for the size L of the reference area inlevel 1 is L=(Threshold) according to FIG. 4, the size L of thereference area in level 1 is constantly nine pixels, as shown in FIG. 5.

Similarly, since the threshold for level 2 is one pixel according toFIG. 4, when the distance is the greatest in level 2 (at the boundarybetween level 2 and level 3), the size of the reference area is onepixel, as shown in FIG. 5. Furthermore, the formula for the size L ofthe reference area in level 2 is L=f₁ (d) according to FIG. 4. d is thedistance from the camera as indicated by the distance information. Thefunction f₁ (d) is a function generated by using an arbitraryinterpolation method, such as Hermite interpolation, linearinterpolation, Lagrange interpolation, or spline interpolation. Aplurality of values (a plurality of pairs of d and L) that are given forthe purpose of calculating the function by using these interpolationmethods are determined by an operator. FIG. 5 shows a function curve inwhich the size of the reference area becomes smaller as the distancefrom the camera becomes longer.

Similarly, since the threshold for level 3 is one pixel according toFIG. 4, when the distance is the greatest in level 3 (at the boundarybetween level 3 and level 4), the size of the reference area is onepixel, as shown in FIG. 5. Furthermore, since the formula for the size Lof the reference area in level 3 is L=(Threshold) according to FIG. 4,the size L of the reference area in level 3 is constantly one pixel, asshown in FIG. 5.

Similarly, since the threshold for level 4 is 25 pixels according toFIG. 4, when the distance is the greatest in level 4 (at the boundarybetween level 4 and level 5), the size of the reference area is 25pixels, as shown in FIG. 5. Furthermore, the formula for the size L ofthe reference area in level 4 is L=f₂(d) according to FIG. 4. Thefunction f₂ (d) is a function generated by using an arbitraryinterpolation method, such as Hermite interpolation, linearinterpolation, Lagrange interpolation, or spline interpolation. Aplurality of values (a plurality of pairs of d and L) that are given forthe purpose of calculating the function by using these interpolationmethods are determined by an operator. FIG. 5 shows a function curve inwhich the size of the reference area becomes larger as the distance fromthe camera becomes longer.

Similarly, since the threshold for level 5 is 25 pixels according toFIG. 4, when the distance is the greatest in level 5, the size of thereference area is 25 pixels, as shown in FIG. 5. Furthermore, since theformula for the size L of the reference area in level 5 is L=(Threshold)according to FIG. 4, the size L of the reference area in level 5 isconstantly 25 pixels, as shown in FIG. 5.

Referring back to FIG. 2, the conversion unit 13 converts the colorinformation on a per-pixel basis on the basis of the settings of themodulation filter (the sizes of reference areas) determined for theindividual pixels by the determination unit 12. Specifically, for eachpixel, the conversion unit 13 sets a reference area having the sizedetermined by the determination unit 12, and converts the colorinformation of that pixel on the basis of the color information of theother pixels in the reference area. For example, the conversion unit 13sets a reference area for each pixel such that the relative positionalrelationship thereof with that pixel becomes a predefined positionalrelationship. The relative positional relationship is, for example, therelationship “each pixel is located at the center of the reference areathereof”, but there is no limitation to this example. There is noparticular limitation to the specific processing for the conversion ofcolor information, and processing using the symmetric nearest neighbor(SNN) filter, the median filter, the Kuwahara filter, or the like may beemployed.

Note that in the case where “one pixel” is set as the size of thereference area, the conversion unit 13 does not convert the colorinformation of that pixel. That is, the color information correspondingto that pixel, acquired by the acquisition unit 11, serves as the colorinformation of the pixel as is, even after the processing for modulatingan image.

Next, an example of the flow of processing by the image processingdevice 10 according to this embodiment will be described with referenceto a flowchart in FIG. 6.

First, the image processing device 10 acquires image informationindicating, on a per-pixel basis, distance information indicating thedistance from the camera and color information (acquisition step S10).For example, the image information is information concerning an imagerepresenting a scene in a game. The distance information, the colorinformation, and the image have been described above in detail. Forexample, during a game, the image processing device 10 can acquire imageinformation concerning an image representing a scene in that game beforethe image is output to a display or the like.

Then, the image processing device 10 determines the settings of themodulation filter that converts the color information of individualpixels to modulate an image into the style of a painting, on a per-pixelbasis on the basis of the distance information of individual pixels(determination step S20). Specifically, as shown in a flowchart in FIG.7, upon acquiring the distance information of individual pixels (S21),the image processing device 10 determines the size of the reference areafor each pixel on the basis of the distance information (S22). Forexample, the image processing device 10 identifies the level whosedistance range includes the distance indicated by the distanceinformation with reference to the reference area information shown inFIG. 4. Then, the image processing device 10 determines the size L ofthe reference area on the basis of the formula for the identified level.The image processing device 10 can execute this processing on aper-pixel basis.

Referring back to FIG. 6, after the determination step S20, the imageprocessing device 10 converts the color information on a per-pixel basison the basis of the settings of the modulation filter (the sizes ofreference areas) determined for the individual pixels in thedetermination step S20 (conversion step S30). Specifically, as shown ina flowchart in FIG. 8, the image processing device 10 sets referenceareas on a per-pixel basis on the basis of the settings of themodulation filter (the sizes of reference areas) determined for theindividual pixels in the determination step S20 (S31). For example, theimage processing device 10 sets a reference area for each pixel suchthat the relative positional relationship thereof with that pixelbecomes a predefined positional relationship. The relative positionalrelationship is, for example, the relationship in which “each pixel islocated at the center of the reference area thereof”, but there is nolimitation to this example. Then, for each pixel, the image processingdevice 10 converts the color information on the basis of the colorinformation of other pixels in the set reference area. There is noparticular limitation to the specific processing for the conversion ofcolor information, and processing using the symmetric nearest neighbor(SNN) filter, the median filter, the Kuwahara filter, or the like may beemployed.

With the image processing device 10 described above, it is possible toset the degrees of conversion (e.g., the degrees of blurring or thedegrees of bleeding) of the color information of individual pixels on aper-pixel basis and to convert the color information of the individualpixels as set on a per-pixel basis. Thus, with the image processingdevice 10, it is possible to vary the degrees of conversion partiallywithin a single image. That is, with the image processing device 10, itis possible to generate a single image in which a portion with a highdegree of conversion, a portion with a low degree of conversion, aportion that has not been converted, etc. coexist. Accordingly, with theimage processing device according to this embodiment, it becomespossible to modulate an image into the style of a painting whilerealizing both edge clarity and painting-like qualities of predeterminedcontent to be rendered.

Furthermore, with the image processing device 10 that executesmodulation processing on the basis of the reference area informationshown in FIGS. 4 and 5, it is possible to determine the sizes ofreference areas having the following features and to execute processingfor modulating an image accordingly.

(Feature 1) In the case where the distance indicated by the distanceinformation is included in the distance range for level 3 (greater thanor equal to D1 and less than or equal to D2), the size of the referencearea is determined to be the size of each pixel (one pixel). That is,modulation into the style of a painting is not executed for an item tobe rendered whose distance from the camera is included in the distancerange.

(Feature 2) In the case where the distance indicated by the distanceinformation is included in the distance range for level 4 (greater thanD2), the reference area is determined so as to have a larger size as thedistance indicated by the distance information becomes longer. That is,for an item to be rendered whose distance from the camera is included inthe distance range, modulation is executed so as to increase the degreeof painting-likeness as the distance from the camera becomes longer.

(Feature 3) In the case where the distance indicated by the distanceinformation is included in the distance range for level 2 (less thanD1), the reference area is determined so as to have a larger size as thedistance indicated by the distance information becomes shorter. That is,for an item to be rendered whose distance from the camera is included inthe distance range, modulation is executed so as to increase the degreeof painting-likeness as the distance from the camera becomes shorter.

For example, the distance range for level 3 is a distance range in whichfocusing is possible. The distance range for level 4 is a distance rangein which the distance from the viewpoint is too long for focusing and inwhich the degree of blurring increases as the distance becomes longer.Meanwhile, the distance range for level 2 is a distance range in whichthe distance from the viewpoint is too short for focusing and in whichthe degree of blurring increases as the distance becomes shorter. On thebasis of the reference area information shown in FIGS. 4 and 5, theimage processing device 10 can modulate an image similar to the wayobjects are actually seen. This makes it possible to generate arealistic image better reflecting the reality.

Now, a modification of the embodiment will be described. In theembodiment described above, the size of the reference area is adopted asa parameter for adjusting the degree of conversion of the colorinformation of each pixel. As the parameter, the color difference or thelike between pixels may be adopted instead of the size of the referencearea. As the color difference increases, the degree of blurring or thedegree of bleeding becomes higher, and the degree of painting-likenessincreases. Meanwhile, as the color difference decreases, the degree ofblurring or the degree of bleeding becomes lower, and the degree ofpainting-likeness decreases. Also with this modification, operations andadvantages similar to those of the embodiment are realized.

Second Embodiment

A game device according to this embodiment includes the functional unitsof the image processing device 10 described in the context of the firstembodiment. FIG. 9 shows an example of the functional block diagram ofthe game device 100. As shown in the figure, the game device 100includes an input accepting unit 1, a player-character control unit 2, anon-player-character control unit 3, a camera control unit 4, arendering unit 5, and a display control unit 6. As will be describedlater in detail, the rendering unit 5 includes the functional units ofthe image processing device 10.

Note that in a game provided by the game device 100 according to thisembodiment, a player character and a non-player character exist in avirtual three-dimensional space. The player character and the non-playercharacter move through the virtual three-dimensional space and engage inbattle with other characters or conversations with other characters. Thevirtual three-dimensional space is represented by an orthogonalcoordinate system defined by the mutually orthogonal X axis, Y axis, andZ axis. Coordinates in the orthogonal coordinate system will hereinafterbe referred to as “coordinates in the virtual three-dimensional space”depending on the case.

The input accepting unit 1 accepts operational inputs from the player.The input accepting unit 1 can accept operational inputs from the playervia any input device, such as a game controller, a pointing device, atouchscreen, a physical button, a mouse, a keyboard, a microphone, or acamera. The specific content of operational inputs is a design matterthat is defined on a per-game basis. For example, the player characterperforms an operational input for causing the player character toperform a predetermined action (e.g., movement).

The player-character control unit 2 controls the player character on thebasis of operational inputs from the player. For example, theplayer-character control unit 2 manages the current position of theplayer character in the virtual three-dimensional space in the form ofcoordinates (x_(p), y_(p), z_(p)) in the virtual three-dimensionalspace. Furthermore, the player-character control unit 2 manages theorientation (e.g., the facing direction) of the player character, forexample, in the form of rotation angles (θ_(xp), θ_(yp), θ_(zp)) aboutthe X axis, the Y axis, and the Z axis with respect to a referencedirection. Furthermore, on the basis of an operational input from theplayer, the player-character control unit 2 determines a new positionand orientation of the player character in the virtual three-dimensionalspace and updates the coordinates (x_(p), y_(p), z_(p)) and rotationangles (θ_(xp), θ_(yp), θ_(zp)).

The non-player-character control unit 3 controls a non-player character.For example, the non-player-character control unit 3 manages the currentposition of the non-player character in the virtual three-dimensionalspace in the form of coordinates (x_(np), y_(np), z_(np)) in the virtualthree-dimensional space. Furthermore, the non-player-character controlunit 3 manages the orientation (e.g., the facing direction) of thenon-player character, for example, in the form of rotation angles(θ_(xnp), θ_(ynp), θ_(znp)) about the X axis, the Y axis, and the Z axiswith respect to a reference direction. Furthermore, by using anarbitrary means (on the basis of a predetermined algorithm), thenon-player-character control unit 3 determines a new position andorientation of the non-player character in the virtual three-dimensionalspace and updates the coordinates (x_(np), y_(np), z_(np)) and rotationangles (θ_(xnp), θ_(ynp), θ_(znp)).

The camera control unit 4 controls the virtual camera located in thevirtual three-dimensional space on the basis of an operational inputfrom the player or a movement of the player character in the virtualthree-dimensional space, arising from an operational input from theplayer. For example, the camera control unit 4 manages the currentposition of the virtual camera in the virtual three-dimensional space inthe form of coordinates (x_(c), y_(c), z_(c)) in the virtualthree-dimensional space. Furthermore, the camera control unit 4 managesthe orientation (e.g., the direction of the optical axis) of the virtualcamera, for example, in the form of rotation angles (θ_(xc), θ_(yc),θ_(zc)) about the X axis, the Y axis, and the Z axis with respect to areference direction. Furthermore, on the basis of an operational inputfrom the player or a movement of the player character in the virtualthree-dimensional space, arising from an operational input from theplayer, the camera control unit 4 determines a new position andorientation of the virtual camera in the virtual three-dimensional spaceand updates the coordinates (x_(c), y_(c), z_(c)) and rotation angles(θ_(xc), θ_(yc), θ_(zc)).

There is no particular limitation to the method of determining theposition and orientation of the virtual camera, and any technology maybe adopted. For example, the virtual camera may be defined so as tophotograph the player character from a position behind the character andseparated by a predetermined distance. In this case, the camera controlunit 4 can change the coordinates (x_(c), y_(c), z_(c)) indicating thecurrent position and the rotation angles (θ_(xc), θ_(yc), θ_(zc))indicating the orientation of the virtual camera so as to follow changesin the coordinates (x_(p), y_(p), z_(p)) indicating the position and therotation angles (θ_(xp), θ_(yp), θ_(zp)) indicating the orientation ofthe player character. Note that although the photographing direction ofthe virtual camera is defined with reference to the player character inthe above example, the photographing direction may be defined withrespect to an object (e.g., a non-player character or an arbitraryobject) different from the player character. Alternatively, the movementpath or rotation direction of the virtual camera may be predefined.Furthermore, the camera control unit 4 may change the current positionand orientation of the virtual camera according to the movement path orrotation direction.

The rendering unit 5 renders the three-dimensional space captured by thevirtual camera as an image on a two-dimensional plane on the basis ofthe current position and orientation of the player character, managed bythe player-character control unit 2, the current position andorientation of the non-player character, managed by thenon-player-character control unit 3, and the current position andorientation of the virtual camera, managed by the camera control unit 4.Note that objects whose positions do not change (e.g., buildings andplants) may exist in the virtual three-dimensional space, andcoordinates indicating the individual positions of the objects in thevirtual three-dimensional space may be registered in advance.Furthermore, the rendering unit 5 may render the image on thetwo-dimensional plane by further using information concerning theobjects.

As shown in FIG. 10, the rendering unit 5 includes an acquisition unit11, a determination unit 12, a conversion unit 13, an informationacquisition unit 14, a two-dimensional-image rendering unit 15, and anoutput unit 16.

The information acquisition unit 14 acquires information (e.g., thecoordinates (x_(p), y_(p), z_(p)) and the rotation angles (θ_(xp),θ_(yp), θ_(zp))) indicating the current position and orientation of theplayer character, managed by the player-character control unit 2,information (e.g., the coordinates (x_(np), y_(np), z_(np)) and therotation angles (θ_(xnp), θ_(ynp), θ_(znp))) indicating the currentposition and orientation of the non-player character, managed by thenon-player-character control unit 3, and information (e.g., thecoordinates (x_(c), y_(c), z_(c)) and the rotation angles (θ_(xc),θ_(yc), θ_(zc))) indicating the current position and orientation of thevirtual camera, managed by the camera control unit 4. The informationacquisition unit 14 may further acquire information indicating thepositions and orientations of objects whose positions do not change.Furthermore, the information acquisition unit 14 may acquire informationindicating the results of various game processing.

On the basis of the information acquired by the information acquisitionunit 14, the two-dimensional-image rendering unit 15 renders thethree-dimensional space captured by the virtual camera as an image on atwo-dimensional plane. There is no particular limitation concerning themethod of realizing this rendering, and any technology may be adopted.For example, the two-dimensional-image rendering unit 15 may executepreprocessing, such as coordinate transformation (world coordinatetransformation or camera coordinate transformation), clippingprocessing, and geometric processing such as perspective transformation.Then, the two-dimensional-image rendering unit 15 may generate renderingdata on the basis of the results of the preprocessing. The renderingdata includes color data, texture coordinates, the coordinates ofvertices on primitive faces, normal vectors, an a value, etc. Then, thetwo-dimensional-image rendering unit 15 generates image informationconcerning an image in which the player character, etc. after theperspective transformation are rendered on the basis of the renderingdata. The image information indicates distance information indicatingthe distance from the virtual camera and color information on aper-pixel basis. Note that the two-dimensional-image rendering unit 15can employ hidden surface removal processing using depth buffering(Z-buffering) or the like.

The acquisition unit 11, the determination unit 12, and the conversionunit 13 execute the processing described in the context of the firstembodiment on the basis of the image information generated by thetwo-dimensional-image rendering unit 15.

The output unit 16 outputs image information indicating an imagegenerated by the two-dimensional-image rendering unit 15 and subjectedto modulation processing by the image processing device 10 (hereinafterreferred to as image information after modulation processing).

Referring back to FIG. 9, the display control unit 6 displays the imageindicated by the image information after modulation processing, outputby the output unit 16 of the rendering unit 5, on an output device.Examples of the output device include a display and a projection device,but there is no limitation to these examples. The output device may bean external device that is physically and logically separated from thegame device 100 and connected to the game device 100. Alternatively, theoutput device may be physically and logically integrated with the gamedevice 100.

An example hardware configuration of the game device 100 is the same asthe example hardware configuration of the image processing device 10described in the context of the first embodiment.

With the game device 100 described above, operations and advantagessimilar to those of the image processing device 10 according to thefirst embodiment are realized.

Note that, as is apparent from the above description, the game device100 renders a scene in the three-dimensional space each time, in whichthe situation constantly changes according to operational inputs fromthe player or under the control of the CPU, instead of rendering a stillspace that is constantly unchanged. Since the situation constantlychanges, the relative positional relationship (distance and orientation)between the virtual camera and an item to be rendered (the playercharacter, etc.) dynamically changes. Under such an environment in whichthe situation constantly changes, the game device 100 can render “thesame item to be rendered” while emphasizing edges in some cases and inthe style of a painting in other cases depending on the situation.

Examples of reference modes are appended below.

-   1. An image processing method,

wherein a computer executes:

-   -   an acquisition step of acquiring image information indicating,        on a per-pixel basis, distance information indicating the        distance from a camera and color information;    -   a determination step of determining, on a per-pixel basis and on        the basis of the distance information of individual pixels,        settings of a modulation filter that converts the color        information of the individual pixels to modulate an image into        the style of a painting; and    -   a conversion step of converting the color information on a        per-pixel basis on the basis of the settings of the modulation        filter determined for the individual pixels.

-   2. An image processing method according to 1,

wherein the computer:

-   -   in the conversion step, sets a reference area for each pixel and        converts the color information of that pixel on the basis of the        color information of the other pixels in the reference area; and    -   in the determination step, determines the size of the reference        area for each pixel on the basis of the distance information.

-   3. An image processing method according to 2,

wherein the computer:

-   -   in the determination step, determines the size of the reference        area for each pixel on the basis of reference area information        and the distance information, the reference area information        defining the size of the reference area on a per-level basis for        N different levels (N is an integer greater than or equal to        two) of the distance from the camera.

-   4. An image processing method according to 3,

wherein the computer:

-   -   in the determination step, determines the size of the reference        area for each pixel on the basis of the reference area        information, which is a formula for calculating the size of the        reference area.

-   5. An image processing method according to any of 2 to 4,

wherein the computer,

-   -   in the determination step,        -   determines the size of each pixel as the size of the            reference area in the case where the distance indicated by            the distance information is included in a range greater than            or equal to D1 and less than or equal to D2;        -   determines the reference area such that the reference area            becomes larger as the distance indicated by the distance            information becomes longer in the case where the distance            indicated by the distance information is greater than D2;            and        -   determines the reference area such that the reference area            becomes larger as the distance indicated by the distance            information becomes shorter in the case where the distance            information is less than D1.

-   6. An image processing device including:

an acquisition unit that acquires image information indicating, on aper-pixel basis, distance information indicating the distance from acamera and color information;

a determination unit that determines, on a per-pixel basis and on thebasis of the distance information of individual pixels, settings of amodulation filter that converts the color information of the individualpixels to modulate an image into the style of a painting; and

a conversion unit that converts the color information on a per-pixelbasis on the basis of the settings of the modulation filter determinedfor the individual pixels.

-   7. A program for causing a computer to function as:

an acquisition means for acquiring image information indicating, on aper-pixel basis, distance information indicating the distance from acamera and color information;

a determination means for determining, on a per-pixel basis and on thebasis of the distance information of individual pixels, settings of amodulation filter that converts the color information of the individualpixels to modulate an image into the style of a painting; and

a conversion means for converting the color information on a per-pixelbasis on the basis of the settings of the modulation filter determinedfor the individual pixels.

-   8. A game device including:

an input accepting unit that accepts an operational input from a player;

a player-character control unit that manages the position andorientation of a player character in a virtual three-dimensional space;

a non-player-character control unit that manages the position andorientation of a non-player character in the virtual three-dimensionalspace;

a camera control unit that manages the position and orientation of avirtual camera in the virtual three-dimensional space;

a rendering unit that renders the virtual three-dimensional spacecaptured by the virtual camera as an image on a two-dimensional plane onthe basis of the positions and orientations of the player character, thenon-player character, and the virtual camera in the virtualthree-dimensional space; and

a display control unit that causes an output device to display the imageon the two-dimensional plane,

wherein the rendering unit includes:

-   -   an acquisition unit that acquires image information indicating,        on a per-pixel basis, distance information indicating the        distance from the virtual camera and color information;    -   a determination unit that determines, on a per-pixel basis and        on the basis of the distance information of individual pixels,        settings of a modulation filter that converts the color        information of the individual pixels to modulate an image into        the style of a painting; and    -   a conversion unit that converts the color information on a        per-pixel basis on the basis of the settings of the modulation        filter determined for the individual pixels.

This application claims priority based on Japanese Patent ApplicationNo. 2018-241536 filed on Dec. 25, 2018, which is incorporated herein inits entirety.

1. An image processing method, wherein a computer executes: anacquisition step of acquiring image information indicating, on aper-pixel basis, distance information indicating the distance from acamera and color information; a determination step of determining, on aper-pixel basis and on the basis of the distance information ofindividual pixels, settings of a modulation filter that converts thecolor information of the individual pixels to modulate an image into thestyle of a painting; and a conversion step of converting the colorinformation on a per-pixel basis on the basis of the settings of themodulation filter determined for the individual pixels.
 2. An imageprocessing method according to claim 1, wherein the computer: in theconversion step, sets a reference area for each pixel and converts thecolor information of that pixel on the basis of the color information ofthe other pixels in the reference area; and in the determination step,determines the size of the reference area for each pixel on the basis ofthe distance information.
 3. An image processing method according toclaim 2, wherein the computer: in the determination step, determines thesize of the reference area for each pixel on the basis of reference areainformation and the distance information, the reference area informationdefining the size of the reference area on a per-level basis for Ndifferent levels (N is an integer greater than or equal to two) of thedistance from the camera.
 4. An image processing method according toclaim 3, wherein the computer: in the determination step, determines thesize of the reference area for each pixel on the basis of the referencearea information, which is a formula for calculating the size of thereference area.
 5. An image processing method according to of claim 2,wherein the computer, in the determination step, determines the size ofeach pixel as the size of the reference area in the case where thedistance indicated by the distance information is included in a rangegreater than or equal to D1 and less than or equal to D2; determines thereference area such that the reference area becomes larger as thedistance indicated by the distance information becomes longer in thecase where the distance indicated by the distance information is greaterthan D2; and determines the reference area such that the reference areabecomes larger as the distance indicated by the distance informationbecomes shorter in the case where the distance indicated by the distanceinformation is less than D1.
 6. An image processing device comprising:an acquisition unit that acquires image information indicating, on aper-pixel basis, distance information indicating the distance from acamera and color information; a determination unit that determines, on aper-pixel basis and on the basis of the distance information ofindividual pixels, settings of a modulation filter that converts thecolor information of the individual pixels to modulate an image into thestyle of a painting; and a conversion unit that converts the colorinformation on a per-pixel basis on the basis of the settings of themodulation filter determined for the individual pixels.
 7. A program forcausing a computer to function as: an acquisition means for acquiringimage information indicating, on a per-pixel basis, distance informationindicating the distance from a camera and color information; adetermination means for determining, on a per-pixel basis and on thebasis of the distance information of individual pixels, settings of amodulation filter that converts the color information of the individualpixels to modulate an image into the style of a painting; and aconversion means for converting the color information on a per-pixelbasis on the basis of the settings of the modulation filter determinedfor the individual pixels.
 8. A game device comprising: an inputaccepting unit that accepts an operational input from a player; aplayer-character control unit that manages the position and orientationof a player character in a virtual three-dimensional space; anon-player-character control unit that manages the position andorientation of a non-player character in the virtual three-dimensionalspace; a camera control unit that manages the position and orientationof a virtual camera in the virtual three-dimensional space; a renderingunit that renders the virtual three-dimensional space captured by thevirtual camera as an image on a two-dimensional plane on the basis ofthe positions and orientations of the player character, the non-playercharacter, and the virtual camera in the virtual three-dimensionalspace; and a display control unit that causes an output device todisplay the image on the two-dimensional plane, wherein the renderingunit includes: an acquisition unit that acquires image informationindicating, on a per-pixel basis, distance information indicating thedistance from the virtual camera and color information; a determinationunit that determines, on a per-pixel basis and on the basis of thedistance information of individual pixels, settings of a modulationfilter that converts the color information of the individual pixels tomodulate an image into the style of a painting; and a conversion unitthat converts the color information on a per-pixel basis on the basis ofthe settings of the modulation filter determined for the individualpixels.